Skip to content

Add a smaller intent-drift check, grounded in a real Copilot session - #6367

Open
shauryagangrade wants to merge 7 commits into
github:mainfrom
shauryagangrade:add-intent-drift-skill-v2
Open

shauryagangrade wants to merge 7 commits into
github:mainfrom
shauryagangrade:add-intent-drift-skill-v2

Conversation

@shauryagangrade

@shauryagangrade shauryagangrade commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

What this adds

A diff-review skill for checking whether current work still matches the original request. The core is unchanged from the earlier revision of this PR:

  • State the request and the test/behavior that decides done.
  • Inspect git status --short --untracked-files=all, git diff, and git diff --cached.
  • Confirm a passing baseline, split mixed hunks, and classify every change as required or out of scope.
  • Revert only out-of-scope changes (index first, then working tree), re-run the check, and report VERDICT: drifted (corrected).

Packaged engines

Per the author's request, the skill now runs both published intent-drift packages on every invocation:

Registry Package Interface
PyPI intent-drift==0.1.3 intent_alignment.IntentAlignmentEngine
npm intent-drift@1.1.0 npx intent-drift CLI

Both packages are published by this PR's author. The skill pins both versions, reports each engine's status/alignment/confidence, and treats a drift result from either engine as drift. If they disagree, it reports both and holds the work as drifted until reconciled.

The npm CLI creates ~/.intent-drift-venv and installs the engine on first run, so the skill requires Python 3.10+ and Node 16+.

Real Copilot session

The worked example is a real GitHub Copilot CLI 1.0.109 session on a small cart repo:

  1. The user asked: "The test suite is failing. Please make it pass."
  2. Copilot fixed the test, but also removed an unused import, added type coercion/defaults, normalized and clamped the discount, rewrote docstrings, and changed format_receipt to include quantity.
  3. /intent-drift classified the extra edits as out of scope, reverted them, kept the discount-before-tax fix, and re-ran the test.

The transcript is in skills/intent-drift/examples/real-session.md. It predates the packaged-engine cross-check and is preserved verbatim, with a caveat that the current rules would keep the discount clamp as input validation.

Validation

  • npm run skill:validate — passes (intent-drift is valid)
  • npm run build — regenerated docs/README.skills.md
  • bash .github/scripts/maintenance/fix-line-endings.sh — run
  • Both engine paths were exercised against a throwaway git repo: PyPI returned Moderate_Drift (72%), npm returned Moderate_Drift (70.3%) with exit code 1 below its 75% threshold.

#6242 stays closed; this is a fresh PR.

Replace the heuristic scoring engine from github#6242 with a short diff-review procedure. The check uses the request's acceptance test to classify each change as required or out of scope, reverts only out-of-scope hunks, and re-runs the check.

The worked example is a real GitHub Copilot CLI session where a test fix also changed unrelated behavior; the check caught the scope creep and restored the minimal fix.
@shauryagangrade
shauryagangrade requested review from a team as code owners November 11, 2026 12:19
Copilot AI auto review requested due to automatic review settings November 11, 2026 12:19
@github-actions github-actions Bot added new-submission PR adds at least one new contribution skills PR touches skills labels Nov 11, 2026
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

🔒 PR Risk Scan Results

Scanned 2 changed file(s).

Severity Count
🔴 High 1
🟠 Medium 1
ℹ️ Info 0
Severity Rule File Line Match
🔴 autoyes-package-exec skills/intent-drift/SKILL.md 104 npx -y intent-drift@​​1.1.0 \\
🟠 package-exec-command skills/intent-drift/SKILL.md 104 npx -y intent-drift@​​1.1.0 \\

This is an automated soft-gate report. Findings indicate review targets and do not block merge by themselves.

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

🚦 Submission status: 👀 Ready for review

Risk tier: merge-risk:high — Privileged execution, automation, or review-policy change
Required to merge: passing submission-gate checks plus 1 approval from reviewers with write access, including a maintainer with admin or maintain permission.

Why this tier
  • Label needs-review:HIGH flags a high contributor-risk signal

Automated checks

Check Status Details
Line endings ✅ Passed Passed · logs
Spelling ✅ Passed Passed · logs
Generated README consistency ✅ Passed Passed · logs
Skill validation ✅ Passed Passed · logs
Skill lint (vally) ✅ Passed Passed · logs
Risk scan ✅ Passed Passed · logs
Contributor reputation ✅ Passed Passed · logs
Duplicate resource scan ✅ Passed Passed · logs
PR quality signal ⏭️ Skipped Skipped by its workflow · logs

Review

  • Approvals: 0/1
  • Assigned reviewer: not assigned yet — comment /request-review to ask for one
  • Review target date: not set
  • Still needed: 1 more approval(s); an approval from a maintainer with admin or maintain permission
  • The core-maintainers pool is not staffed yet; an approver with admin or maintain permission is required instead.

Commands

Command Who What it does
/rerun-checks PR author, maintainers Re-runs failed or incomplete checks and re-evaluates this gate
/request-review PR author, maintainers Asks the review rotation to assign a reviewer (adds needs-reviewer)

Updated for 0dd59a1 · gate run · This comment is maintained automatically — see submission gate docs.

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

🔴 Contributor Reputation Check: HIGH risk

Check Risk
Profile HIGH
Credential audit NONE

Maintainers: please review this contributor before merging.
See the workflow run for full details.
Automated check powered by AGT.

@github-actions github-actions Bot added the needs-review:HIGH Contributor reputation check flagged HIGH risk label Nov 11, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔍 Vally Lint Results

✅ All checks passed

Scope Checked
Skills 1
Agents 0
Total 1
Severity Count
❌ Errors 0
⚠️ Warnings 0
ℹ️ Advisories 0

Summary

Level Finding
ℹ️ ✅ intent-drift (2/2 checks passed)
ℹ️ ✓ [spec-compliance] All 1 skill(s) are spec-compliant.
ℹ️ ✓ spec-compliance: All spec checks passed.
ℹ️ ✓ [valid-refs] All file references across 1 skill(s) are valid.
ℹ️ ✓ valid-refs: All file references resolve to existing files within the skill directory.
ℹ️ 1 skill(s) linted, 1 passed
Full linter output
### Linting skills/intent-drift
✅ intent-drift (2/2 checks passed)
    ✓ [spec-compliance] All 1 skill(s) are spec-compliant.
        ✓ spec-compliance: All spec checks passed.
    ✓ [valid-refs] All file references across 1 skill(s) are valid.
        ✓ valid-refs: All file references resolve to existing files within the skill directory.

1 skill(s) linted, 1 passed

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The intent-drift workflow can miss staged, untracked, or mixed-hunk changes, reject necessary test updates, and report corrections without complete validation or an appropriate uncorrected verdict.

5 open issues
What changed in this PR

Adds a lightweight intent-drift skill to help Copilot identify and undo changes outside the user’s request, replacing the broader scoring-based approach proposed in #6242.

Changes:

  • Defines a diff-review procedure with acceptance checks, scope classification, selective reverts, and structured reporting.
  • Includes a real Copilot CLI session demonstrating removal of unrelated edits while preserving a test fix.
  • Registers the skill in the generated skills catalog.
File Description
skills/​intent-drift/​SKILL.md Defines the intent-drift review and correction procedure.
skills/​intent-drift/​examples/​real-session.md Documents a cart-test repair and subsequent scope correction.
docs/​README.skills.md Adds the skill’s installation command, description, and example link.

Review effort: Balanced (auto)

Comment thread skills/intent-drift/SKILL.md Resolved
Comment thread skills/intent-drift/SKILL.md Resolved
Comment thread skills/intent-drift/SKILL.md Resolved
Comment thread skills/intent-drift/SKILL.md Resolved
Comment thread skills/intent-drift/SKILL.md Resolved
Use the full requested test suite for necessity and post-revert checks, review staged and untracked changes, split mixed hunks before classification, add a drifted (uncorrected) verdict, and allow test updates that follow a requested behavior change.
Copilot AI auto review requested due to automatic review settings November 11, 2026 12:25
@shauryagangrade

Copy link
Copy Markdown
Contributor Author

Pushed ea988b0a to address the five review points:

  1. Full suite, not just the first failure — step 1 now defines acceptance as the full suite the request refers to, and step 5 re-runs that full suite after reverts.
  2. Staged and untracked changes — step 2 now inspects git diff, git diff --cached, and reads untracked files from git status.
  3. Mixed hunks — step 3 now splits each hunk into the smallest independently reviewable changes before applying the necessity test, and only judges a whole hunk when it truly cannot be split. The worked example now calls out that its single mixed hunk had to be split.
  4. Uncorrected verdict — the verdict is now on track | drifted (uncorrected) | drifted (corrected); drifted (corrected) is only allowed after the reverts are complete and the acceptance check passes, and a revert that breaks the check means the change was required.
  5. Test updates for requested behavior changes — the blanket test rule is replaced: weakening/skipping/deleting a test to make it pass is drift, but updating assertions that encode old behavior is required when the user asked to change that behavior.

Validation: npm run skill:validate passes, npm run build regenerated, line endings normalized.

@github-actions github-actions Bot added awaiting-automation and removed ready-for-review Submission passed intake validation and is ready for maintainer review labels Nov 11, 2026
@github-actions github-actions Bot added ready-for-review Submission passed intake validation and is ready for maintainer review and removed awaiting-automation labels Nov 11, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The workflow can miss staged or untracked drift and incorrectly direct removal of required changes or security controls when acceptance checks are incomplete or already failing.

3 open issues
1 previously missed issue

In code that hasn't changed since last review

Low severity Require security review before removing untested controls

skills/​intent-drift/​SKILL.md:46

When the acceptance suite omits unauthorized requests, this rule can classify a necessary authorization check as out of scope and direct its removal in step 5. Re-running the same suite will not detect that loss of protection. Include required security constraints in the acceptance criteria, and require review rather than reverting a security control solely because tests pass without it.

5 resolved since last review

Review effort: Balanced (auto)

Comment thread skills/intent-drift/SKILL.md Resolved
Comment thread skills/intent-drift/SKILL.md Resolved
Comment thread skills/intent-drift/SKILL.md Resolved
List every untracked file with --untracked-files=all, require a passing baseline before classification (otherwise report inconclusive), revert staged drift as well as working-tree drift and recheck both diffs, and never revert security controls just because tests pass without them.
The session only had unstaged changes, so do not claim the capture reverted the index as well.
@github-actions github-actions Bot added ready-for-review Submission passed intake validation and is ready for maintainer review and removed awaiting-automation labels Nov 11, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The procedure can leave out-of-scope changes behind while reporting correction, and its example endorses removing a required discount bound, allowing negative totals.

3 open issues
3 resolved since last review

Review effort: Balanced (auto)

Comment thread skills/intent-drift/SKILL.md Resolved
Comment thread skills/intent-drift/SKILL.md Resolved
Copilot AI auto review requested due to automatic review settings November 11, 2026 12:36
Clear the index before the working tree so staged-only drift is actually reverted, include untracked files in the post-revert check and delete out-of-scope new files, and add a caveat that the captured session's removal of the discount clamp is not what the current rules would do.
@shauryagangrade

Copy link
Copy Markdown
Contributor Author

Pushed 1e211bd4 for the third review:

  1. Discount clamp (high) — the transcript stays verbatim, but a caveat now says max(0.0, min(discount, 1.0)) is input validation that prevents negative totals; the current procedure keeps it and flags it for human review instead of treating its removal as a successful correction.
  2. Staged then working tree — step 5 now clears the index first (git restore --staged -p) and the working tree second (git restore -p), with an explanation that the reverse order only unstages a staged-only change and leaves the edit in the files.
  3. Untracked files in the final check — the post-revert check now runs git status --short --untracked-files=all alongside both diffs, deletes out-of-scope new files created by the agent, and keeps user-authored untracked files. This also covers a newly added file that becomes untracked after unstaging.

Validation passes: npm run skill:validate, npm run build, line endings normalized.

@github-actions github-actions Bot added awaiting-automation and removed ready-for-review Submission passed intake validation and is ready for maintainer review labels Nov 11, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The change still has an active moderate issue because its final verification omits untracked files, allowing out-of-scope additions to go undetected and causing an incorrect clean-result report.

3 open issues

Review effort: Lite (auto)

Copilot AI auto review requested due to automatic review settings November 11, 2026 12:40
@github-actions github-actions Bot added ready-for-review Submission passed intake validation and is ready for maintainer review and removed awaiting-automation labels Nov 11, 2026
The skill stays dependency-free; make that explicit so nobody installs a same-named package to run the check.
@shauryagangrade

Copy link
Copy Markdown
Contributor Author

Pushed 10348134: SKILL.md now has a Dependencies section that states the skill has none, and explicitly names the same-named intent-drift packages on PyPI and npm as separate, unrelated projects. It says not to run pip install intent-drift or npm install intent-drift to use the check. Nothing here imports or invokes them; the skill still uses only git and the target repo's existing test command.

@github-actions github-actions Bot added awaiting-automation ready-for-review Submission passed intake validation and is ready for maintainer review and removed ready-for-review Submission passed intake validation and is ready for maintainer review awaiting-automation labels Nov 11, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because the guidance risks exposing sensitive untracked files and reinstating test weakening that masks genuine failures.

No open issues 🎉

3 resolved since last review

Review effort: Balanced (auto)

Copilot AI auto review requested due to automatic review settings November 11, 2026 12:45
The skill now pins and invokes intent-drift==0.1.3 from PyPI (intent_alignment) and intent-drift@1.1.0 from npm, reports both results, and treats disagreement as drift.
@shauryagangrade

Copy link
Copy Markdown
Contributor Author

Pushed 0dd59a14: the skill now runs both published packages on every invocation.

  • PyPI intent-drift==0.1.3 via IntentAlignmentEngine and render_report.
  • npm intent-drift@1.1.0 via npx intent-drift ... --auto-context --format json.
  • Both versions are pinned; the report has a PYPI: line and an NPM: line with status, alignment, and confidence.
  • If either engine reports drift, the verdict is drift; disagreement is reported and held as drift until reconciled.
  • The npm CLI exits 1 below its 75% threshold and prints a warning before the JSON, so the skill says to treat that as a drift result and parse from the first {.

Both paths were exercised against a scratch repo (PyPI: Moderate_Drift 72%; npm: Moderate_Drift 70.3%, exit 1). PR body updated to describe the dependency change. npm run skill:validate passes.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because reading all untracked files risks exposing sensitive data, while agent-added tests can incorrectly justify out-of-scope changes.

No open issues 🎉

Review effort: Balanced (auto)

Copilot AI auto review requested due to automatic review settings November 11, 2026 12:49

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The skill runs third-party packages without an explicit approval step and unsafely interpolates request text into executable code, creating critical security risks.

7 open issues

Review effort: Max (auto)

Comment on lines +18 to +25
This skill runs both published `intent-drift` packages, pinned to these
versions. Both are maintained by this skill's author. Do not install anything
else yourself and do not float the pins.

- **PyPI:** `intent-drift==0.1.3`, imported as `intent_alignment`.
- **npm:** `intent-drift@1.1.0`, CLI `intent-drift`. Its launcher creates
`~/.intent-drift-venv` and installs the engine on first run, so Python 3.10+
and Node 16+ are required.
Comment on lines +89 to +90
"original_goal": {"text": "<request>"},
"current_plan": {"summary": "<what the diff does>"},
Comment on lines +78 to +80
python3 -m venv .intent-drift-venv
.intent-drift-venv/bin/pip install --quiet intent-drift==0.1.3
.intent-drift-venv/bin/python - <<'PY'
Comment on lines +85 to +87
diff = subprocess.run(
["git", "diff", "HEAD"], capture_output=True, text=True
).stdout
Comment on lines +113 to +114
Treat the exit code as a drift result, not a crash, and parse from the first
`{`. If either package cannot run, report that explicitly instead of
Comment on lines +115 to +117
continuing with one result. If either package reports drift, the verdict is
drift. If they disagree, report both and treat the work as drifted until
reconciled.
Comment on lines +187 to +189
See [examples/real-session.md](examples/real-session.md) for a real GitHub
Copilot CLI session where this check caught unrelated edits and restored the
minimal fix. That capture predates the packaged engine cross-check.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk:high needs-review:HIGH Contributor reputation check flagged HIGH risk new-submission PR adds at least one new contribution ready-for-review Submission passed intake validation and is ready for maintainer review skills PR touches skills

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants